Kernel methods and regularization techniques for nonparametric regression: Minimax optimality and adaptation

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We first derive risk bounds for these techniques that match the minimax rates in several settings, using recent large deviations machinery and a natural bias-variance decomposition. We then show that certain regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression.